From pgvector/pgvector README

With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average. For more rows, increase hnsw.ef_search.

Starting with 0.8.0, you can enable iterative index scans, which will automatically scan more of the index when needed.

Since we are stuck on 0.7.0 we are going the first option for now.